Linguistic Dumpster Diving: Geographical Classification of Arabic Text

In many applied natural language processing tasks, information is thrown out. For example, in speech recognition systems, prosodic information is commonly discarded; in information retrieval systems, a document is commonly treated as an unordered bag of words and syntactic information is thrown out; and in machine translation systems, pragmatic information (e.g., topic-comment structure and referents of anaphoric expressions) is commonly discarded. Perhaps the most common discarded linguistic forms are the frequent words of a language—words such as those shown in figure 1.